Modern computing systems have a general need to transfer large quantities of data from one machine to another, often over long distances. It has long been a problem in computing that transferring large quantities of data is a time consuming and costly process. Conventional systems have addressed the problems of data transfer in several ways. One technique includes data compression. Generally, data compression involves encoding information using fewer bits than the original representation. Lossless compression reduces bits by identifying and eliminating statistical redundancy, while lossy compression reduces bits by identifying marginally important information and removing it. Compression is useful because it helps reduce resource usage, such as data storage space or transmission capacity. However, compression involves a significant amount of data analysis and this extra processing imposes computational or other costs.
Another approach includes increasing the available bandwidth used to transfer the data. Bandwidth sometimes defines the net bit rate (or peak bit rate, information rate or physical layer useful bit rate), channel capacity, or the maximum throughput of a logical or physical communication path in a digital communication system. Thus, the speed of data transfer over the communication path is limited by the bandwidth. Increasing the bandwidth, however, may be very expensive, as it comes with the cost of interrupting or delaying other data transfers or adding to or even replacing the communication path.
A third technique used to address the problems of data transfer is the physical transport of storage media. In this case, the data one wishes to transfer is stored on storage media (e.g., one or more optical disks), which is physically delivered to the location of a second computer and loaded onto that computer. This can be a cheap method of data transfer with potentially a very high bandwidth, however, it can take significantly longer than transferring data in an electronic signal either wirelessly or over a wire, resulting in a very high latency.
Often times a user of a computing system must determine whether to place greater emphasis on the time it takes to transfer the data or the cost of transferring the data in a shorter amount of time. Many techniques in use today suffer adversely in one area or the other.